Systems and methods for registration of a medical device using a reduced search space

ABSTRACT

A method comprises identifying connected anatomical structures in stored images of a patient anatomy and generating a first three-dimensional model of anatomic passageways including a set of anatomical centerlines of the anatomic passageways, and generating a second three-dimensional model of the anatomic passageways including a plurality of linkage elements each including two base points defining a linkage element centerline spanning a shortest distance between the two base points. The method also comprises receiving tracking data corresponding to a point on an instrument positioned within the connected anatomical structures, matching the point on the instrument to one of the plurality of linkage elements representing one of the connected anatomical structures to locate the instrument relative to the connected anatomical structures in the stored images of the patient anatomy, and updating the matching of the point on the instrument to one of the plurality of linkage elements.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application61/646,654 filed May 14, 2012, which is incorporated by reference hereinin its entirety.

FIELD

The present disclosure is directed to systems and methods for tracking amedical device within a patient anatomy during a medical procedure, andmore particularly to systems and methods for efficiently tracking amedical device within a patient anatomy using a reduced search space.

BACKGROUND

Minimally invasive medical techniques are intended to reduce the amountof tissue that is damaged during diagnostic or surgical procedures,thereby reducing patient recovery time, discomfort, and deleterious sideeffects. Such minimally invasive techniques may be performed throughnatural orifices in a patient anatomy or through one or more surgicalincisions. Through these natural orifices or incisions clinicians mayinsert surgical instruments to reach a target tissue location. To reachthe target tissue location, the minimally invasive surgical instrumentsmay navigate natural or surgically created connected passageways inanatomical systems such as the lungs, the colon, the intestines, thekidneys, the heart, the circulatory system, or the like. Navigationalassist systems help the clinician route the surgical instruments andavoid damage to the anatomy. These systems can incorporate the use ofposition and shape sensors to more accurately describe the shape, pose,and location of the surgical instrument in real space or with respect topre-procedural or concurrent images. In a dynamic anatomical systemand/or in an anatomical region dense with many anatomical passageways,accurately registering a minimally invasive instrument to the anatomicalsystem is a time consuming and computer processing intensive task.Improved systems and methods are needed for increasing the accuracy andefficiency of systems and methods of registering minimally invasiveinstruments to the anatomical system.

SUMMARY

The embodiments of the invention are summarized by the claims thatfollow below.

In one embodiment, a method comprises identifying connected anatomicalstructures in stored images of a patient anatomy and generating aplurality of linkage elements representing the connected anatomicalstructures. The method also comprises receiving tracking datacorresponding to a point on an instrument positioned within at least oneof the connected anatomical structures; and matching the point on theinstrument to one of the plurality of linkage elements.

In another embodiment, a medical system comprises a flexible instrumentincluding a sensor adapted to provide tracking data for a point on theinstrument and a memory storing images of a patient anatomy. The systemalso comprises a processor that identifies connected anatomicalstructures in the stored images of the patient anatomy and generates aplurality of linkage elements representing the connected anatomicalstructures. The processor also receives the tracking data correspondingto the point on an instrument positioned within at least one of theconnected anatomical structures and matches the point on the instrumentto one of the plurality of linkage elements.

In another embodiment, a method comprises generating an image of a setof connected passageways of a patient anatomy and receiving positioninformation for a point on an instrument indicating a position of thepoint relative to the connected passageways. The method also includesgenerating adjusted position information for the point includingmodeling the set of connected passageways of a patient anatomy as astructure of linked cylinders and creating an adjusted instrument imagewith the adjusted position information. The method also includesgenerating a composite image including the image of the set of connectedpassageways and the adjusted instrument image.

BRIEF DESCRIPTIONS OF THE DRAWINGS

Aspects of the present disclosure are best understood from the followingdetailed description when read with the accompanying figures. It isemphasized that, in accordance with the standard practice in theindustry, various features are not drawn to scale. In fact, thedimensions of the various features may be arbitrarily increased orreduced for clarity of discussion. In addition, the present disclosuremay repeat reference numerals and/or letters in the various examples.This repetition is for the purpose of simplicity and clarity and doesnot in itself dictate a relationship between the various embodimentsand/or configurations discussed.

FIG. 1 is a robotic surgical system, in accordance with embodiments ofthe present disclosure.

FIG. 2 illustrates a surgical instrument system utilizing aspects of thepresent disclosure.

FIG. 3a is an image of a human lung registered with an image of aminimally invasive instrument.

FIG. 3b is an internal image of the human lung depicting the region ofthe human lung from the viewpoint of the minimally invasive instrument.

FIG. 4 is an illustration of a portion of bronchial passageway of alung.

FIG. 5 is an illustration of a plurality of cylindrical linkage elementsrepresenting the bronchial passageway of FIG. 4.

FIG. 6 is an illustration of a cylindrical linkage element.

FIG. 7 is a flowchart illustrating a method for snapping a point of aninstrument to an anatomical passageway.

FIG. 8 is an illustration of a cylindrical linkage element.

FIG. 9 is an illustration of an instrument point projected to acenterline of a cylindrical linkage element.

DETAILED DESCRIPTION

In the following detailed description of the embodiments of theinvention, numerous specific details are set forth in order to provide athorough understanding of the disclosed embodiments. However, it will beobvious to one skilled in the art that the embodiments of thisdisclosure may be practiced without these specific details. In otherinstances well known methods, procedures, components, and circuits havenot been described in detail so as not to unnecessarily obscure aspectsof the embodiments of the invention.

The embodiments below will describe various instruments and portions ofinstruments in terms of their state in three-dimensional space. As usedherein, the term “position” refers to the location of an object or aportion of an object in a three-dimensional space (e.g., three degreesof translational freedom along Cartesian X,Y,Z coordinates). As usedherein, the term “orientation” refers to the rotational placement of anobject or a portion of an object (three degrees of rotationalfreedom—e.g., roll, pitch, and yaw). As used herein, the term “pose”refers to the position of an object or a portion of an object in atleast one degree of translational freedom and to the orientation of thatobject or portion of the object in at least one degree of rotationalfreedom (up to six total degrees of freedom). As used herein, the term“shape” refers to a set of poses, positions, or orientations measuredalong an object.

Referring to FIG. 1 of the drawings, a robotic surgical system isgenerally indicated by the reference numeral 100. As shown in FIG. 1,the robotic system 100 generally includes a surgical manipulatorassembly 102 for operating a surgical instrument 104 in performingvarious procedures on the patient P. The assembly 102 is mounted to ornear an operating table O. A master assembly 106 allows the surgeon S toview the surgical site and to control the manipulator assembly 102.

In alternative embodiments, the robotic system may include more than onemanipulator assembly. The exact number of manipulator assemblies willdepend on the surgical procedure and the space constraints within theoperating room among other factors.

The master assembly 106 may be located at a surgeon's console C which isusually located in the same room as operating table O. However, itshould be understood that the surgeon S can be located in a differentroom or a completely different building from the patient P. Masterassembly 106 generally includes an optional support 108 and one or morecontrol device(s) 112 for controlling the manipulator assemblies 102.The control device(s) 112 may include any number of a variety of inputdevices, such as joysticks, trackballs, gloves, trigger-guns,hand-operated controllers, voice recognition devices or the like. Insome embodiments, the control device(s) 112 will be provided with thesame degrees of freedom as the associated surgical instruments 104 toprovide the surgeon with telepresence, or the perception that thecontrol device(s) 112 are integral with the instruments 104 so that thesurgeon has a strong sense of directly controlling instruments 104. Insome embodiments, the control devices 112 are manual input devices whichmove with six degrees of freedom, and which may also include anactuatable handle for actuating instruments (for example, for closinggrasping jaws, applying an electrical potential to an electrode,delivering a medicinal treatment, or the like).

A visualization system 110 may include a viewing scope assembly(described in greater detail below) such that a concurrent or real-timeimage of the surgical site is provided to surgeon console C. Theconcurrent image may be, for example, a two or three dimensional imagecaptured by an endoscope positioned within the surgical site. In thisembodiment, the visualization system 100 includes endoscopic componentsthat may be integrally or removably coupled to the surgical instrument104. However in alternative embodiments, a separate endoscope, attachedto a separate manipulator assembly may be used with the surgicalinstrument to image the surgical site. The visualization system 110 maybe implemented as hardware, firmware, software or a combination thereofwhich interact with or are otherwise executed by one or more computerprocessors, which may include the processors of a control system 116(described below).

A display system 111 may display an image of the surgical site andsurgical instruments captured by the visualization system 110. Thedisplay 111 and the master control devices 112 may be oriented such thatthe relative positions of the imaging device in the scope assembly andthe surgical instruments are similar to the relative positions of thesurgeon's eyes and hands so the operator can manipulate the surgicalinstrument 104 and the hand control as if viewing the workspace insubstantially true presence. By true presence, it is meant that thepresentation of an image is a true perspective image simulating theviewpoint of an operator that is physically manipulating the surgicalinstruments 104.

Alternatively or additionally, monitor 111 may present images of thesurgical site recorded and/or modeled preoperatively using imagingtechnology such as, computerized tomography (CT), magnetic resonanceimaging (MRI), fluoroscopy, thermography, ultrasound, optical coherencetomography (OCT), thermal imaging, impedence imaging, laser imaging, ornanotube X-ray imaging. The presented preoperative images may includetwo-dimensional, three-dimensional, or four-dimensional images.

In some embodiments, the monitor 111 may display a virtual navigationalimage in which the actual location of the surgical instrument isregistered (i.e., dynamically referenced) with preoperative orconcurrent images to present the surgeon S with a virtual image of theinternal surgical site at the location of the tip of the surgicalinstrument. An image of the tip of the surgical instrument or othergraphical or alphanumeric indicators may be superimposed on the virtualimage to assist the surgeon controlling the surgical instrument.Alternatively, the surgical instrument may not be visible in the virtualimage.

In other embodiments, the monitor 111 may display a virtual navigationalimage in which the actual location of the surgical instrument isregistered with preoperative or concurrent images to present the surgeonS with a virtual image of surgical instrument within the surgical sitefrom an external viewpoint. An image of a portion of the surgicalinstrument or other graphical or alphanumeric indicators may besuperimposed on the virtual image to assist the surgeon controlling thesurgical instrument.

As shown in FIG. 1, a control system 116 includes at least one processorand typically a plurality of processors for effecting control betweenthe surgical manipulator assembly 102, the master assembly 106, and theimage and display system 110. The control system 116 also includessoftware programming instructions to implement some or all of themethods described herein. While control system 116 is shown as a singleblock in the simplified schematic of FIG. 1, the system may comprise anumber of data processing circuits (e.g., on the surgical manipulatorassembly 102 and/or on the master assembly 106), with at least a portionof the processing optionally being performed adjacent an input device, aportion being performed adjacent a manipulator, and the like. Any of awide variety of centralized or distributed data processing architecturesmay be employed. Similarly, the programming code may be implemented as anumber of separate programs or subroutines, or may be integrated into anumber of other aspects of the robotic systems described herein. In oneembodiment, control system 116 may support wireless communicationprotocols such as Bluetooth, IrDA, HomeRF, IEEE 802.11, DECT, andWireless Telemetry.

In some embodiments, control system 116 may include servo controllers toprovide force and torque feedback from the surgical instruments 104 tothe hand-operated control device 112. Any suitable conventional orspecialized servo controller may be used. A servo controller may beseparate from, or integral with manipulator assemblies 102. In someembodiments, the servo controller and manipulator assembly are providedas part of a robotic arm cart positioned adjacent to the patient's body.The servo controller transmits signals instructing the manipulatorassemblies to move instruments which extend into an internal surgicalsite within the patient body via openings in the body.

Each of the manipulator assemblies 102 that support a surgicalinstrument 104 and may comprise a series of manually articulatablelinkages, generally referred to as set-up joints, and a roboticmanipulator. The robotic manipulator assemblies 102 may be driven by aseries of actuators (e.g., motors). These motors actively move therobotic manipulators in response to commands from the control system116. The motors are further coupled to the surgical instrument so as toadvance the surgical instrument into a naturally or surgically createdanatomical orifice and to move the distal end of the surgical instrumentin multiple degrees of freedom that may include three degrees of linearmotion (e.g., X, Y, Z linear motion) and three degrees of rotationalmotion (e.g., roll, pitch, yaw). Additionally, the motors can be used toactuate an articulatable end effector of the instrument for graspingtissues in the jaws of a biopsy device or the like.

FIG. 2 illustrates a tracked instrument system 118 which includes thesurgical instrument system 104 and its interfacing systems. The surgicalinstrument system 104 includes a flexible instrument 120 coupled by aninterface 122 to manipulator assembly 102 and visualization system 110.The instrument 120 has a flexible body 124, a tip 126 at its distal end128, and the interface 122 at its proximal end 130. The body 124 housescables, linkages, or other steering controls (not shown) that extendbetween the interface 122 and the tip 126 to controllably bend or turnthe tip as shown for example by the dotted line versions of the bent tip126, and in some embodiments control an optional end effector 132. Theflexible instrument may be steerable, including the steering controlspreviously described, or may be non-steerable with no integratedmechanism for operator control of the instrument bending. The endeffector may be a working distal part that is manipulable for a medicalfunction, e.g., for effecting a predetermined treatment of a targettissue. For instance, some end effectors have a single working membersuch as a scalpel, a blade, or an electrode. Other end effectors such asshown in the embodiment of FIG. 2, have a pair or plurality of workingmembers such as forceps, graspers, scissors, or clip appliers, forexample. Examples of electrically activated end effectors includeelectrosurgical electrodes, transducers, sensors, and the like. Endeffectors may also include conduits to convey fluids, gases or solids toperform, for example, suction, insufflation, irrigation, treatmentsrequiring fluid delivery, accessory introduction, biopsy extraction andthe like). In other embodiments, flexible body 124 can define one ormore lumens through which surgical instruments can be deployed and usedat a target surgical location.

The instrument 120 can also include an image capture element 134 whichmay include a stereoscopic or monoscopic camera disposed at the distalend 128 for capturing images that are transmitted to and processed bythe visualization system 110 for display by the display system 111.Alternatively, the image capture element 134 may be a coherentfiber-optic bundle that couples to an imaging and processing system onthe proximal end of the instrument 120, such as a fiberscope. The imagecapture element 134 may be single or multi-spectral for capturing imagedata in the visible or infrared/ultraviolet spectrum.

A tracking system 135 includes an electromagnetic (EM) sensor system 136and a shape sensor system 138 for determining the position, orientation,speed, pose, and/or shape of the distal end 128 and of one or moresegments 137 along the instrument 120. The tracking system may also beused for registering the stored images of the patient anatomy and theshape sensor in a reference frame fixed to the patient. Registrationmay, alternatively, be performed using other techniques that do notutilize the EM sensor system, as would be understood by a person ofordinary skill in the art. Although only an exemplary set of segments137 are depicted in FIG. 2, the entire length of the instrument 120,between the distal end 128 and the proximal end 130 and including thetip 126 may be effectively divided into segments. The tracking system135 may be implemented as hardware, firmware, software or a combinationthereof which interact with or are otherwise executed by one or morecomputer processors, which may include the processors of a controlsystem 116.

The EM sensor system 136 includes one or more conductive coils that maybe subjected to an externally generated electromagnetic field. Each coilof the EM sensor system 136 then produces an induced electrical signalhaving characteristics that depend on the position and orientation ofthe coil relative to the externally generated electromagnetic field. Inone embodiment, the EM sensor system may be configured and positioned tomeasure six degrees of freedom, e.g., three position coordinates X, Y, Zand three orientation angles indicating pitch, yaw, and roll of a basepoint. Further description of an EM sensor system is provided in U.S.Pat. No. 6,380,732, filed Aug. 11, 1999, disclosing “Six-Degree ofFreedom Tracking System Having a Passive Transponder on the Object BeingTracked,” which is incorporated by reference herein in its entirety.

The sensor system 138 includes an optical fiber 140 aligned with theflexible body 124 (e.g., provided within an interior channel (not shown)or mounted externally). The tracking system 135 is coupled to a proximalend of the optical fiber 140. In this embodiment, the fiber 140 has adiameter of approximately 200 μm. In other embodiments, the dimensionsmay be larger or smaller.

The optical fiber 140 forms a fiber optic bend sensor for determiningthe shape of the instrument 120. In one alternative, optical fibersincluding Fiber Bragg Gratings (FBGs) are used to provide strainmeasurements in structures in one or more dimensions. Various systemsand methods for monitoring the shape and relative position of a opticalfiber in three dimensions are described in U.S. patent application Ser.No. 11/180,389, filed Jul. 13, 2005, disclosing “Fiber optic positionand shape sensing device and method relating thereto;” U.S. ProvisionalPat. App. No. 60/588,336, filed on Jul. 16, 2004, disclosing“Fiber-optic shape and relative position sensing;” and U.S. Pat. No.6,389,187, filed on Jun. 17, 1998, disclosing “Optical Fibre BendSensor,” which are incorporated by reference herein in their entireties.In other alternatives, sensors employing other strain sensing techniquessuch as Rayleigh scattering, Raman scattering, Brillouin scattering, andFluorescence scattering may be suitable. In other alternativeembodiments, the shape of the instrument 120 may be determined usingother techniques. For example, if the history of instrument tip's poseis stored for an interval of time that is smaller than the period forrefreshing the navigation display or for alternating motion (e.g.,inhalation and exhalation), the pose history can be used to reconstructthe shape of the device over the interval of time. As another example,historical pose, position, or orientation data may be stored for a knownpoint of an instrument along a cycle of alternating motion, such asbreathing. This stored data may be used to develop shape informationabout the instrument. Alternatively, a series of positional sensors,such as EM sensors, positioned along the instrument can be used forshape sensing. Alternatively, a history of data from a positionalsensor, such as an EM sensor, on the instrument during a procedure maybe used to represent the shape of the instrument, particularly if ananatomical passageway is generally static. Alternatively, a wirelessdevice with position or orientation controlled by an external magneticfield may be used for shape sensing. The history of its position may beused to determine a shape for the navigated passageways.

In this embodiment, the optical fiber 140 may include multiple coreswithin a single cladding 146. Each core may be single-mode withsufficient distance and cladding separating the cores such that thelight in each core does not interact significantly with the lightcarried in other cores. In other embodiments, the number of cores mayvary or each core may be contained in a separate optical fiber.

In some embodiments, an array of FBG's is provided within each core.Each FBG comprises a series of modulations of the core's refractiveindex so as to generate a spatial periodicity in the refraction index.The spacing may be chosen so that the partial reflections from eachindex change add coherently for a narrow band of wavelengths, andtherefore reflect only this narrow band of wavelengths while passingthrough a much broader band. During fabrication of the FBG's, themodulations are spaced by a known distance, thereby causing reflectionof a known band of wavelengths. However, when a strain is induced on thefiber core, the spacing of the modulations will change, depending on theamount of strain in the core. Alternatively, backscatter or otheroptical phenomena that vary with bending of the optical fiber can beused to determine strain within each core.

Thus, to measure strain, light is sent down the fiber, andcharacteristics of the returning light are measured. For example, FBG'sproduce a reflected wavelength that is a function of the strain on thefiber and its temperature. This FBG technology is commercially availablefrom a variety of sources, such as Smart Fibres Ltd. of Bracknell,England. Use of FBG technology in position sensors for robotic surgeryis described in U.S. Pat. No. 7,930,065, filed Jul. 20, 2006, disclosing“Robotic Surgery System Including Position Sensors Using Fiber BraggGratings,” which is incorporated by reference herein in its entirety.

When applied to a multicore fiber, bending of the optical fiber inducesstrain on the cores that can be measured by monitoring the wavelengthshifts in each core. By having two or more cores disposed off-axis inthe fiber, bending of the fiber induces different strains on each of thecores. These strains are a function of the local degree of bending ofthe fiber. For example, regions of the cores containing FBG's, iflocated at points where the fiber is bent, can thereby be used todetermine the amount of bending at those points. These data, combinedwith the known spacings of the FBG regions, can be used to reconstructthe shape of the fiber. Such a system has been described by LunaInnovations. Inc. of Blacksburg, Va.

As described, the optical fiber 140 is used to monitor the shape of atleast a portion of the instrument 120. More specifically, light passingthrough the optical fiber 140 is processed by the tracking system 135for detecting the shape of the surgical instrument 120 and for utilizingthat information to assist in surgical procedures. The tracking system135 may include a detection system for generating and detecting thelight used for determining the shape of the instrument 120. Thisinformation, in turn, in can be used to determine other relatedvariables, such as velocity and acceleration of the parts of a surgicalinstrument. By obtaining accurate measurements of one or more of thesevariables in real time, the controller can improve the accuracy of therobotic surgical system and compensate for errors introduced in drivingthe component parts. The sensing may be limited only to the degrees offreedom that are actuated by the robotic system, or may be applied toboth passive (e.g., unactuated bending of the rigid members betweenjoints) and active (e.g., actuated movement of the instrument) degreesof freedom.

The information from the tracking system 135 may be sent to thenavigation system 142 where it is combined with information from thevisualization system 110 and/or the preoperatively taken images toprovide the surgeon or other operator with real-time positioninformation on the display system 111 for use in the control of theinstrument 120. The control system 116 may utilize the positioninformation as feedback for positioning the instrument 120. Varioussystems for using fiber optic sensors to register and display a surgicalinstrument with surgical images are provided in U.S. patent applicationSer. No. 13/107,562, filed May 13, 2011, disclosing, “Medical SystemProviding Dynamic Registration of a Model of an Anatomical Structure forImage-Guided Surgery,” which is incorporated by reference herein in itsentirety.

In the embodiment of FIG. 2, the instrument 104 is teleoperated withinthe robotic surgical system 100. In an alternative embodiment, themanipulator assembly may be replaced by direct operator control. In thedirect operation alternative, various handles and operator interfacesmay be included for hand-held operation of the instrument.

FIG. 3a depicts a composite image 150 including an image 151 of a humanlung 152, from a viewpoint external to the lung, registered with aninstrument image 154 of a flexible instrument, such as the flexibleinstrument 120. The image 151 of the lung 152 may be generated frompreoperatively recorded images or may be generated concurrently duringthe surgical procedure. The composite image 150 may be displayed viadisplay system 111. As the instrument 120 is advanced through bronchialpassageways 156 of the lung 152, information from the tracking system135 and/or the visualization system 110 are used to register theinstrument image 154 with the lung image 151. The image 151 of the lung152 may change, for example, to depict the lung in a state ofinspiration or expiration. The instrument image 154 may change to depictthe advancement or withdrawal of the instrument 120 through thebronchial passageways 156. Occasionally, the composite image 150 mayerroneously render the instrument image 154 such that a portion of theinstrument image 154′ is outside of a bronchial passageway. Systems andmethods are described below for correcting the instrument image suchthat the instrument is located within the bronchial passageways.

FIG. 3b depicts an internal image 160 of the human lung 152 depicting aregion of the lung from the viewpoint of the instrument 120. The image160 may be a concurrent image taken during the surgical procedure by theinstrument 120 while located in the depicted portion of the lung 152.More specifically, the image may be captured by the visualization system110. Alternatively, the image 160 may be a preoperatively recorded imageselected based upon the location of the tip of the instrument 120 asdetermined by the tracking system 135.

Often, the tracking system 135, which includes the EM sensor system 136and the shape sensor system 138 may calculate a position for theinstrument tip 126 or one or more segments 137 of the instrument 120that is outside the anatomical passageway. This likely indicates aslight measurement error (assuming the wall of the anatomical passagewayhas not been breached). Such an error may result from the dynamic natureof certain anatomic structures such as the lungs or the heart. Forexample, inhalation and exhalation changes the position and size of thebronchial passageways of the lung. Alternatively, the error may resultfrom tissue deformation caused by the presence of the surgicalinstrument within the anatomic passageways. To correct the position ofthe instrument and accurately locate one or more points of theinstrument within the passageway when the image of the instrument andthe image of the patient anatomy are co-registered and displayed,selected points of the instrument may be snapped or graphicallyregistered to a location on the wall of the anatomical passageway or tothe lumen of the anatomical passageway.

In one embodiment snapping a point on an instrument to an anatomicalpassageway begins by modeling a stored image of the anatomicalpassageway. FIG. 4 is a three dimensional model 200 of a portion ofbronchial structure 202 of a lung as captured in preoperative image. Thebronchial structure 202 includes discrete bronchial passageways 204-216.A series of centerpoints through the bronchial passageways 204-216 formthe anatomic centerline 218. The image 200 may be a composite imageformed from a plurality of preoperative images. FIG. 5 is anillustration 300 of a three dimensional linked structure 302 ofcylindrical linkage elements 304-316 representing the bronchialpassageways 204-216. As will be described in detail below, the connectedcylindrical linkage elements 304-316 are determined from the anatomiccenterline 218. Representing the bronchial structure 202 as the linkedcylinder structure 302 may reduce the search space for determining thespecific bronchial passageway in which an instrument point is located.Although reference is frequently made to branched anatomicalpassageways, the systems and embodiments of this application are alsoapplicable to modeling a single passageway as a series of joinedcylinders. This may be particularly useful if the passageway is curvedor varies in diameter along its length.

FIG. 6 is an illustration of the cylindrical linkage element 304. Thecylindrical linkage element 304 is formed about a branch 320 of anatomiccenterline 218. The branch 320 includes a base point BP0 and a basepoint BP1 at opposite ends of the branch. A cylindrical centerline 322spans the shortest distance between the base point BP0 and the basepoint BP1. The total radius R_(T) of the cylindrical linkage element 304can be the combined length of radius R₁ and radius R₂. The radius R₁ isthe maximum deviation between the anatomic centerline 218 and thecylindrical centerline 322. Thus, the larger the deviation between thecenterlines, the larger the total cylinder radius approximation. Theradius R₂ is the average radius of the bronchial passageway 204 asobtained, for example, from the model 200. In one embodiment, thecylindrical linkage element 304 can include spherically-shaped ends 324,326. The overlapping spherically-shaped ends 324, 326 can simplify thecomputational burden associated with modeling linkage element 304, andcan provide an intuitive “joint” element with adjacent linkage elements.However, in various other embodiments, cylindrical linkage element 304can have ends of any configuration (e.g., flat, angled, faceted, or anyother modeling configuration).

Note that in various other embodiments, the total radius R_(T) can bedefined in various alternative manners to provide a desired level ofrepresentation for branch 320. For example, in some embodiments, totalradius R_(T) can be simply be equivalent to the average radius of thebronchial passageway 204 (i.e., radius R2), the maximum radius ofbronchial passageway 204, or any other appropriate characteristic ofbronchial passageway 204. In other embodiments, radius R₁ (or any otheradjustment factor such as a pre-determined “buffer” distance, or even avarying radius along cylindrical linkage element 304) can be added toany of the above bronchial passageway-related radii.

FIG. 7 is a flowchart 400 illustrating a method for snapping a threedimensional point P of the instrument 154 to a bronchial passageway.This method may be used to correct the image of the instrument at pointP when, for example as shown in FIG. 3a , the tracking systeminformation, including information from a shape sensor and/or an EMsensor, positions a portion of the instrument, including point P,outside of a bronchial passageway. Initially, connected anatomicalstructures are identified in stored images of a patient anatomy.Connected anatomical structures include branching anatomicalpassageways, where each passageway is connected to at least one otherpassageway. A plurality of cylindrical linkage elements are generated torepresent the connected anatomical structures, as described above.

At step 402, a point P is identified to be snapped to a bronchialpassageway. The point P may have been identified as a point on abronchial instrument that has been tracked to a location outside of abronchial passageway. In some embodiments the point P may be a distaltip of the bronchial instrument, but in alternative embodiments thepoint P may be any point along the instrument within the bronchialpassageway. At step 404, the closest of the cylindrical linkage elements304-316 to point P are identified. For example, to determine thedistance between the point P and each cylindrical linkage element304-316, the general location of the point P relative to the cylindricallinkage element is first determined. The general location of the point Prelative to each of the cylindrical linkage elements is consideredbecause the distance to be calculated is the distance between the pointP and the closest point on the cylindrical centerline of each element,including the base points BP0 and BP1.

With reference to FIG. 8, the point P may be in one of three locationsrelative to the cylindrical linkage element 304. First, the point Pcould be P_(A), in the region A near BP0. Second, the point P could beP_(B), in the region B between BP0 and BP1. Third, the point P could beP_(C), in the region C near BP1. To determine the region in which pointP is located, the inner product of the unit directional vector V fromBP0 to BP1 and the vector from BP0 to P is calculated. If the innerproduct is less than zero, P is closest to BP0 (region A). If the innerproduct is between BP0 and BP1, P is closer to BP1 (region C). If theinner product is otherwise, P is somewhere at a midpoint between BP0 andBP1. After the location of the point P relative to the cylindricallinkage element 304 is determined, the distance from P to the closestpoint on the centerline 322 is calculated. This determination of point Plocation and distance is performed for each of the cylindrical linkageelements 304-316. The cylindrical linkage elements 304-316 with theshortest distances to P are considered the closest cylindrical linkageelements. Further processing is performed only on these closestcylindrical linkage elements. In one embodiment, the closest fivecylindrical linkage elements may be chosen, but in other embodimentsmore or fewer elements may be suitable.

At step 406, for the closest cylindrical linkage elements to point P, asearch is performed to identify the closest point on the anatomiccenterline for each of the closest cylindrical linkage elements. Afterthe closest point on the anatomic centerline is determined for each ofthe closes cylindrical linkage elements, the cylindrical linkage elementwith the closest overall anatomic centerline point P_(i) is chosen.

At step 408 and with reference to FIG. 9, a final point P_(f) along theanatomic centerline of the closest cylindrical linkage element isdetermined. More specifically, refinement calculations are performed todetermine if final point P_(f) is between two centerline point P_(i) andP_(i+1) or between Pi and P_(i−1). After the final point P_(f) isdetermined, the point P is projected along a straight line towards P_(f)until it coincides with a sphere around P_(f) with a radiusapproximately equal to the diameter of the bronchial passage at P_(f).The point P_(S) represents the position on the wall of the bronchialpassageway to which the point P will be snapped.

At step 410, the point P is evaluated to determine whether it is outsidea cylindrical linkage element or inside a cylindrical linkage element.If the point P is inside the cylindrical linkage element, at step 412,the rendered image of the point P on the instrument image is maintainedwithin the bronchial passage on the co-registered lung image. If thepoint P is outside the cylindrical linkage element, at step 414, therendered image of the point P on the instrument image is adjusted orsnapped to the point P_(S) along the projection between P and P_(f)where the projection intersects the wall of the bronchial passage. Atstep 416, a corrected image composite image 150 depicting the image 151of a human lung 152 registered with an corrected instrument image 154 isprepared. In the corrected images, the point P is snapped to thebronchial passageway rather than extending outside the bronchial wall.

The systems and methods described may conserve time and computerprocessing resources as compared to “brute-force” methods that attemptto directly snap a point P to the correct bronchial passageway bycomprehensively considering each point along the wall of a bronchialpassageway or at each point along a centerline of the bronchialpassageway. For example, the time for the snapping procedure with themethods of this disclosure may be approximately 35 μs compared to 210 μsfor the brute-force method.

Alternative systems and methods for registering an image of at least aportion of a flexible instrument to an image of an anatomical system aredisclosed in U.S. Pat. No. 10,085,671, filed May 13, 2013, disclosing“Systems and Methods for Deformation Compensation Using Shape Sensing”and in U.S. Pat. App. No. 13/892,924, filed May 13, 2013, disclosing“Systems and Methods for Registration of a Medical Device Using RapidPose Search,” both of which are incorporated by reference herein intheir entirety. Aspects of theses incorporated by reference systems andmethods may be used together with the above disclosed systems andmethods to provide alternative methods of accurately registering animage of at least a portion of a flexible instrument to an image of ananatomical system.

Although the registration systems and methods have been described hereinwith respect to teleoperated or hand operated surgical systems, theseregistration systems and methods will find application in a variety ofmedical and non-medical instruments in which accurate instrument imageregistration is otherwise too time consuming or computer processingintensive.

Although the systems and methods of this disclosure have beenillustrated for use in the connected bronchial passageways of the lung,they are also suited for navigation and treatment of other tissues, vianatural or surgically created connected passageways, in any of a varietyof anatomical systems including the colon, the intestines, the kidneys,the brain, the heart, the circulatory system, or the like. The methodsand embodiments of this disclosure are also suitable for non-surgicalapplications.

One or more elements in embodiments of the invention may be implementedin software to execute on a processor of a computer system such ascontrol system 116. When implemented in software, the elements of theembodiments of the invention are essentially the code segments toperform the necessary tasks. The program or code segments can be storedin a processor readable storage medium or device that may have beendownloaded by way of a computer data signal embodied in a carrier waveover a transmission medium or a communication link. The processorreadable storage device may include any medium that can storeinformation including an optical medium, semiconductor medium, andmagnetic medium. Processor readable storage device examples include anelectronic circuit; a semiconductor device, a semiconductor memorydevice, a read only memory (ROM), a flash memory, an erasableprogrammable read only memory (EPROM); a floppy diskette, a CD-ROM, anoptical disk, a hard disk, or other storage device, The code segmentsmay be downloaded via computer networks such as the Internet, Intranet,etc.

Note that the processes and displays presented may not inherently berelated to any particular computer or other apparatus. Variousgeneral-purpose systems may be used with programs in accordance with theteachings herein, or it may prove convenient to construct a morespecialized apparatus to perform the operations described. The requiredstructure for a variety of these systems will appear as elements in theclaims. In addition, the embodiments of the invention are not describedwith reference to any particular programming language. It will beappreciated that a variety of programming languages may be used toimplement the teachings of the invention as described herein.

While certain exemplary embodiments of the invention have been describedand shown in the accompanying drawings, it is to be understood that suchembodiments are merely illustrative of and not restrictive on the broadinvention, and that the embodiments of the invention not be limited tothe specific constructions and arrangements shown and described, sincevarious other modifications may occur to those ordinarily skilled in theart.

Although the embodiments described above use cylindrical linkages tomodel the anatomic structure, a person of skill in the art would knowhow to use additional bounding-surface linkages with well-definedsurfaces, without departing from the spirit of the invention. Forexample, ellipsoidal, hyperbolic, spheroidal linkages could be used. Ingeneral, well-defined surfaces of bounding envelope type could be usedfor any of the embodiments above.

What is claimed is:
 1. A method of registering a medical instrument tostored images of a patient anatomy, the method comprising: identifying,by a processor, connected anatomical structures in stored images ofinterior structures of the patient anatomy; generating, by theprocessor, a first three-dimensional model of anatomic passagewaysincluded in the connected anatomical structures, the firstthree-dimensional model including a set of anatomical centerlines of theanatomic passageways; generating, by the processor, a secondthree-dimensional model of the anatomic passageways, the secondthree-dimensional model including a plurality of linkage elements witheach linkage element including two base points disposed on a firstanatomical centerline of the set of anatomical centerlines, the two basepoints defining a linkage element centerline spanning a shortestdistance between the two base points; receiving tracking data from atracking system, the tracking data corresponding to a point on themedical instrument while positioned within the connected anatomicalstructures, the medical instrument being sized for insertion within theconnected anatomical structures; matching, by the processor, the pointon the medical instrument to one of the plurality of linkage elementsrepresenting one of the connected anatomical structures to locate themedical instrument relative to the connected anatomical structures inthe stored images of the patient anatomy, wherein the matching includes:determining, from the second three-dimensional model, a subset of theplurality of linkage elements closer to the point on the medicalinstrument than each linkage element in a remainder of the plurality oflinkage elements based on a plurality of vectors between the point onthe medical instrument and one of the two base points of each of theplurality of linkage elements; and for each linkage element of thesubset, determining a distance between the point on the medicalinstrument and the respective anatomical centerline extending throughthe linkage element; and updating the matching of the point on themedical instrument to one of the plurality of linkage elements as themedical instrument is moved within the anatomic passageways.
 2. Themethod of claim 1 further comprising: receiving preoperative medicalimages into a computer memory device to create the stored images of thepatient anatomy; and displaying, in a display system, one stored imageof the stored images of the patient anatomy, wherein the one storedimage is selected for display based on a location of the medicalinstrument as determined based on the matching of the point on themedical instrument to the one of the plurality of linkage elements. 3.The method of claim 1 wherein the two base points are derived from acenterline model of the connected anatomical structures.
 4. The methodof claim 1 further comprising identifying a closest linkage element inthe subset.
 5. The method of claim 4 further comprising creating aprojection of the point on the medical instrument to the linkage elementcenterline extending within the closest linkage element.
 6. The methodof claim 5 further comprising identifying a snapping point along theprojection at a radial distance from the linkage element centerline. 7.The method of claim 5 further comprising determining whether the pointon the medical instrument is closest to one of the two base points ofthe linkage element centerline through the closest linkage element or isclosest to a midpoint between the two base points along the linkageelement centerline.
 8. A medical system comprising: a flexibleinstrument including a sensor adapted to provide tracking data for apoint on the flexible instrument while the flexible instrument isdisposed within connected anatomical structures of a patient anatomy,the flexible instrument being sized for positioning within the connectedanatomical structures; a memory storing images of the patient anatomy;and a processor configured for: identifying the connected anatomicalstructures in the stored images of the patient anatomy to generate a setof anatomical centerlines characterizing the connected anatomicalstructures; generating, from the set of anatomical centerlines, aplurality of linkage elements representing the connected anatomicalstructures, each linkage element including two base points disposed onan anatomical centerline from the set of anatomical centerlines, the twobase points defining a linkage element centerline extendingtherebetween, wherein the linkage element centerline is different thanthe anatomical centerline in that the linkage element centerline is astraight line between the two base points; receiving the tracking datacorresponding to the point on the flexible instrument positioned withinat least one of the connected anatomical structures; matching the pointon the flexible instrument to one of the plurality of linkage elementsby: determining a plurality of vectors between the point on the flexibleinstrument and one of the two base points of each of the plurality oflinkage elements; identifying a subset of the plurality of linkageelements in a region of the point on the flexible instrument based onthe plurality of vectors; and for each element of the subset of theplurality of linkage elements, determining a distance between the pointon the flexible instrument and a respective anatomical centerline; andupdating the matched point on the flexible instrument to another of theplurality of linkage elements as the flexible instrument is moved withinthe connected anatomical structures.
 9. The medical system of claim 8wherein the stored images of the patient anatomy include preoperativelyrecorded medical images.
 10. The medical system of claim 8 each of theplurality of linkage elements includes two points derived from acenterline model of the connected anatomical structures.
 11. The medicalsystem of claim 8 wherein the processor is further configured toidentify a closest linkage element in the subset.
 12. The medical systemof claim 11 wherein the processor is further configured to create aprojection of the point on the flexible instrument to the linkageelement centerline associated with the closest linkage element.
 13. Themedical system of claim 12 wherein the processor is further configuredto identify a snapping point along the projection at a radial distancefrom the linkage element centerline.
 14. The medical system of claim 12wherein the processor is further configured to determine whether thepoint on the flexible instrument is closest to one of the two basepoints of the anatomical centerline through the closest linkage elementor is closest to a midpoint on the linkage element centerline betweenthe two base points.
 15. The medical system of claim 8 wherein theprocessor is further configured to generate a composite image includingan image of the connected anatomical structures and an image of thepoint on the flexible instrument within the patient anatomy.
 16. Themethod of claim 1 wherein each linkage element defines a volume based onthe two base points and the linkage element centerline.
 17. The methodof claim 16 wherein the volume is defined by a radius having a firstradial component based on a maximum deviation between the linkageelement centerline and an anatomical centerline between the two basepoints.
 18. The method of claim 16 wherein the volume is defined by aradius that varies along the linkage element centerline.
 19. The medicalsystem of claim 8 wherein the flexible instrument further includes anelectromagnetic sensor system having one or more conductive coilsdisposed along a flexible body of the flexible instrument.
 20. Themedical system of claim 8 wherein the processor is configured forreceiving the tracking data from a fiber optic bend sensor to determinecoordinates of the point on the flexible instrument while positionedwithin the at least one of the connected anatomical structures.
 21. Themethod of claim 1, wherein the determining of the subset of theplurality of linkage elements includes, for each linkage element in theplurality of linkage elements, comparing a vector between a respectivetwo base points and a vector of the plurality of vectors that is betweenone of the respective two base points and the point on the medicalinstrument.
 22. The medical system of claim 8 wherein the processor isfurther configured to, for each linkage element in the plurality oflinkage elements, compare a vector extending from a first of arespective two base points to a second of the respective two base pointswith a vector of the plurality of vectors that extends from the first ofthe respective two base points to the point on the flexible instrument.23. The method of claim 4 further comprising determining whether thepoint on the medical instrument is outside the closest linkage element.24. The method of claim 6, wherein the identifying of the snapping pointis based on determining that the point on the medical instrument isoutside the closest linkage element.